A need has arisen for both the users and network operators to have a set of mechanisms to monitor network performance, detect router availability, and troubleshoot network congestion without introducing additional traffic on any communication network. This is especially relevant to Internet providers that must comply with SLAs (Service Level Agreements) that they provide to customers. As Internet architecture evolves, the SLAs now include requirements on the quality of service such as jitter, throughput, one-way packet delay, and packet loss ratio. Additionally, the need to monitor the network traffic is prevalent for the underlying Internet protocol enabling the World Wide Web. Accordingly, various types of mechanisms for network monitoring have been developed lately, as described next.
Active network performance monitoring systems are based on sending a series of special test packets or query packets to the underlying networks or routers, and analyzing the response with a specific purpose. Currently, most tools for active end-to-end QoS monitoring in IP networks are based on the traditional “ping” (i.e., ICMP echo and echo response messages) to measure the roundtrip delay between two hosts. Although these additional packets may provide information about the network performance, they do not provide information about the duration of the traffic flows.
Passive network performance monitoring mechanisms perform traffic analysis in a non-invasive way with respect to the observed networking environment. As a result, these mechanisms do not affect the performance of the network while doing the measurements and querying. A traditional such approach usually involves collecting the entire TCP/IP packet or packet header data and analyzing the collected information followed by off-line traffic characterization and modeling; flow start and stop time may also be recorded. However, this approach is infeasible at high traffic speeds, and also requires large amounts of memory and processing capacity.
Various examples of passive mechanisms have been implemented by certain entities. For example, Cisco offers the NetFlow traffic analyzer that identifies traffic flows based on IP source/destination addresses, protocol ID field, type of service field, and router port. Once identified, statistics can be collected for a traffic flow, and exported via user datagram protocol (UDP) when the flow expires. A NetFlow record contains information about flows that pass through the router and provides a digest of the communications showing hosts that were involved, services that were used, and how much data was exchanged.
As another example, Lucent Bell Labs has various research projects in traffic analysis, which are mainly concentrated on collection of TCP/UDP/IP packet headers and off-line traffic analysis, modeling and visualization.
Another method of collecting network statistics is described in “Estimating flow distributions from sampled flow statistics” Duffield et al, available at http://public.research.att.com/˜duffield/papers/DLT03-lengths.pdf. This paper provides methods that use flow statistics formed from a sampled packet stream to infer the frequencies of the number of packets per flow in the unsampled packet stream. This is obtained by using statistical inference, and exploiting protocol level detail reported in flow records.
However, while the passive network performance monitoring mechanisms described above may provide information about the network performance, they do not provide information about the duration of the traffic flows. Generally, these mechanisms only track the length or size of flows (e.g. the number of packets), which is relatively easy to determine in a sampled environment, but they do not determine the flow duration; currently, the inventor is not aware of any reliable method for tracking flow duration.
As such, the current state of sampled network monitoring solutions remains basic, providing limited information to service providers. Much of the information which is currently not available, like flow duration, can be essential in tracking down anomalous activity in a network.
Tracking flow duration is particularly relevant for network vendors who wish to provide access to information on their high-end routers; they must therefore devise scalable and efficient algorithms to deal with the limited per-packet processing time available. Network operators can use flow duration information to implement cost saving measures, and detect high-cost network traffic such as point-to-point traffic. Detailed visibility into individual users and business applications using the global network is essential for optimizing performance and delivering network services to business users.
There is a need to provide a method of determining when a traffic flow was established with a high degree of accuracy for enabling network operators/providers to perform traffic engineering, particularly in networks with high-end routers, where the per-packet processing time is limited. To address this need the focus of the present invention is to create a method of tracking the duration of every flow in a sampled environment, while maintaining low resource usage.